Exploring rater judging behaviour using the many-facet rasch model

Performance assessment, unlike the traditional fixed-response assessment, has features peculiar to its assessment setting (the task choice, the task processing conditions, the raters,the rating scale, and the rating procedures) that make it much more vulnerable to construct irrelevant variance (McNamara, 1997; Upshur & Turner, 1999). Of these potential sources of variability, those associated with raters are considered to be extensive and pose serious threats to the validity of ratings (Linacre, 1989; McNamara, 1996).For performance assessment to yield valid and reliable results, it is essential that these sources of variability are eliminated or minimized. This paper illustrates how sources of rater-related variability or rater effects can be identified and controlled for using the Many-facet Rasch Model.It also illustrates how the idiosyncratic rating behaviour of individual raters can be explicated through the use of this measurement model. In this illustration, the ratings of thirty-five English language instructors on 12 paragraphs written by new intake students at the Centre for Foundation Studies, IIUM were analysed using FACETS (Linacre, 2003), a computer application which implements the Many-facet Rasch Model. The results of the analysis indicate substantial differences in rater severity, and the presence of other rater effects

Using The Rasch Measurement Model For Standard Setting Of The English Language Placement Test At The IIUM [LB1631. L714 2007 f rb].

Dengan penggunaan skor sempadan dan standad untuk membuat keputusan-keputusan pendidikan yang berciri “high-stakes”, pelbagai usaha seharusnya dibuat untuk mencari kaedah-kaedah penetapan standad yang tidak dipertikaikan. With the use of cut scores and standards for making high-stakes educational decisions, efforts should be made to search for more defensible standard setting methods

Using The Rasch Measurement Model For Standard Setting Of The English Language Placement Test At The Iium

With the use of cutscores and standards for making high-stakes educational decisions, efforts should be made to search for more defensible standard setting methods

Mapping students’ performance in English reading literacy using the Rasch Measurement Model

Monitoring learners’ performance in learning areas over time and across grade levels is essential to ensure that they are improving and achieving expected levels. In the Malaysian context, the monitoring and evaluation of students’ learning and achievement are conducted primarily through national standardized examinations such as the PMR (Penilaian Menengah Rendah/Lower Secondary Examination) and SPM (Sijil Pelajaran Malaysia/Malaysian Certificate of Examination).Though information gained from these examinations is useful, it is limited to students’ learning at particular levels of schooling. These examinations are also norm-referenced, making it impossible to determine the skills and knowledge that have been and have not been acquired. This paper highlights the utility of the Rasch Measurement Model (RMM) in the monitoring students’ performance across grade levels in the context of English reading literacy, and mapping students’ learning progress. A sample of 944 Form 1, 2, and 3 students was selected from eleven national-type secondary schools in two states in Malaysia. A test of English reading ability, comprising of 60 items, was used. RMM was used to produce person-maps and item-skill maps to clearly indicate what students have achieved and what has yet to be achieved. The results indicated that students differed in their performance across grade levels with lower levels showing higher achievement. The findings indicate the need for monitoring students’ progress over time and the use of a robust measurement model

Testing the Imperforate Anus Psychosocial Questionnaire (IAPSQ) on a Sample of Egyptian Children

Imperforate anus is a malformation of the anus, children with such type of congenital malformation usually undergo reconstructive surgery and be looked after by pediatric surgery nurses due to the problems with constipation and fecal incontinence after the colostomy is closed. Lack of control over bowels may affect a child’s psychosocial adjustment. The Aims of This Study are: (a) To evaluate the psychometric properties of the Arabic version of the Imperforate Anus Psychosocial Questionnaire (IAPSQ) on a sample of Egyptian children. (b) To assess the effect of (IA deformity, hypospadias, and rectal prolapse) on the psychosocial functioning of Egyptian children. (c) To compare the psychosocial functioning of children with congenital anomalies of the anus with those suffering from chronic diseases. Methodology: Subjects and Methods Study sample consisted of 35 children (with congenital anomalies of the anus) from Mansoura University Children’s Hospital (n=30), and Alexandria University Hospital (n=5), in Egypt. Data Collection: The data set of the current study was obtained within 3 months (from January to March 2017). Data Processing, Statistical Analysis and Results: Rasch analysis was used to evaluate the psychometric properties of the IAPSQ, as well as to generate the item difficulty and person ability measures. Analysis of the psychological dimension (23 items) gave a low person reliability (.37). With regard to the social dimension (12 items), the Rasch analysis showed a person reliability of .54. Comparisons between the Swedish Sample and the Egyptian Sample Psychological Domain: The Egyptian sample scored this item higher, indicating greater perceived concern by their mothers. it is more of a concerned for the Swedish sample compared to the Egyptian sample. Fathers are perceived by the Egyptian sample to be less loving and thinking more about their conditions compared to their mothers. Social Domain: Children distribution for this domain is also well spread out similar to the Swedish sample. Comparison between the two groups indicated that children with imperforate anus malformation has a higher quality of life (mean = 2.26 logits) compared to those children with chronic disease (mean = 1.78 logits). Reliability of measures is also higher for the study group (.76) compared to the comparison group (.41). The results of the DIF analysis showed that no significant DIF across the study and control group

A construct validation of the Malaysian University English Test and the English Placement Test: Two high-stakes English language proficiency tests

Malaysian students are normally required to show a certain level of English language proficiency before they are offered a place by a local institution of higher learning. One of the examinations conducted to ascertain their level of proficiency is the Malaysian University English Test (MUET). Apart from MUET some universities also conduct their own test for the same reason. This paper discusses a study on the comparability of MUET with the English Placement Test (EPT) conducted by the International Islamic University Malaysia (IIUM). This study was conducted to see whether the two tests could be used interchangeably for admission and placement purposes. The two tests were administered on second year pre university students of IIUM. Analyses show that though significant, the correlation coefficients of the two tests were not high enough for them to be used interchangeably

Factorial validity and invariance of the Muet writing rating scale: empirical and theoretical correspondence

The validity of performance assessment depends to a large extent on the rating scale that is used in the assessment procedure. It delineates the theoretical construct being measured and yield considerable influence on how performances are judged and interpreted. Implications of the use of rating scales are therefore far too important to be taken lightly, particularly in high-stakes standardized tests. This study investigates the factorial validity and invariance of a writing rating scale used in such a test – the Malaysian University English Test – in terms of its correspondence to empirical data and the theoretical construct of ESL writing through the use of confirmatory factor analysis (CFA). To test for factorial validity, a measurement model was developed based on the construct definition of the MUET rating scale and tested using AMOS, a data-fitting programme